Text input device and method therefor

ABSTRACT

Electronic device includes display, microphone, and processor configured to activate voice input function based on user input, display graphic representation for indicating that the voice input function is activated, provide, on the display, a text display area for displaying text inputted by a plurality of user input methods and a keyboard input interface for receiving a user keyboard input, the plurality of user input methods including user keyboard input method and user voice input method, receive, via the keyboard input interface, the user keyboard input corresponding to a first text, display the first text in the text display area based on receiving the user keyboard input, receive user voice input corresponding to a second text while the keyboard input interface is provided and the voice input function is activated, and display the second text next to the first text in the text display area based on the user voice input.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a Continuation of U.S. application Ser. No. 17/093,148, filedNov. 9, 2020, which is Continuation of International Application No.PCT/KR2019/007521, which was filed on Jun. 21, 2019 and claims priorityto Korean Patent Application No. 10-2018-0071817, filed on Jun. 22,2018. The contents of the above-named applications are incorporated byreference herein in their entireties.

BACKGROUND 1. Field

The present disclosure relates to a device and method for inputting atext in response to a voice input.

2. Description of Related Art

With the development of information technology (IT), electronic devices,such as smartphones, which perform various functions in combination havebeen developed, and electronic devices having a voice recognitionfunction have been released in order to improve operability. A voicerecognition function has an advantage that enables a device to be easilycontrolled by recognizing a user's voice without contacting a separateoperation button or a touch module.

According to a voice recognition function, for example, a portableterminal such as a smartphone may execute a call function, write a textmessage, and easily set various functions such as pathfinding, Internetsearch, or alarm setting, without pressing a separate button.

There is a need for a text input device and method that provide a voiceinput interface capable of efficiently recognizing a user voice input.

SUMMARY

A text input device according to an embodiment of the present disclosureincludes: a voice input interface configured to recognize a voice input;a script input interface configured to recognize a script input; and aprocessor configured to determine whether a voice activation conditionindicating whether a user's voice input is started is satisfied,activate the voice input interface based on a result of the determining,and simultaneously provide the activated voice input interface and thescript input interface.

A text input method according to an embodiment of the present disclosureincludes: providing a script input interface configured to recognize ascript input; determining whether a voice activation conditionindicating whether a user's voice input is started is satisfied; andactivating a voice input interface configured to recognize a voiceinput, based on a result of the determining, and simultaneouslyproviding the activated voice input interface and the script inputinterface.

A computer-readable recording medium according to an embodiment of thepresent disclosure may be a computer-readable recording medium havingrecorded thereon a program for executing the above-described method.

According to an embodiment, a text input device and method that providea voice input interface recognizing a user voice input may be provided.

BRIEF DESCRIPTION OF DRAWINGS

The present disclosure may be easily understood by combining thefollowing detailed description and the accompanying drawings, andreference numerals refer to structural elements.

FIG. 1 is a diagram of a text input device according to an embodiment.

FIG. 2 is a diagram of a text input device according to an embodiment.

FIG. 3 is a diagram of a text input device according to an embodiment.

FIG. 4A is a diagram for describing a method, performed by a text inputdevice, of performing a text input operation, according to someembodiments.

FIG. 4B is a diagram for describing a method, performed by a text inputdevice, of performing a text input operation, according to someembodiments.

FIG. 5 is a diagram for describing a method, performed by a text inputdevice, of performing a text input operation based on contextinformation, according to some embodiments.

FIG. 6 is a flowchart for describing a text input method according to anembodiment.

FIG. 7 is a flowchart for describing a method of determining a voiceactivation condition, according to an embodiment.

FIG. 8 is a flowchart for describing a method of determining a voiceactivation condition, according to an embodiment.

FIG. 9 is a flowchart for describing a method of determining a voiceactivation condition, according to an embodiment.

FIG. 10 is a flowchart for describing a method of determining a voiceactivation condition, according to an embodiment.

FIG. 11 is a flowchart for describing a method of determining a voiceactivation condition, according to an embodiment.

FIG. 12 is a flowchart for describing a text input method according toan embodiment.

FIG. 13 is a block diagram of a first device according to someembodiments.

FIG. 14 is a block diagram of a first device according to someembodiments.

FIG. 15 is a block diagram of a processor according to some embodiments.

FIG. 16 is a block diagram of a data trainer according to someembodiments.

FIG. 17 is a block diagram of a data recognizer according to someembodiments.

FIG. 18 is a diagram illustrating an example in which a text inputdevice and a server interoperate with each other to train and recognizedata, according to some embodiments.

DETAILED DESCRIPTION

A text input device according to an embodiment of the present disclosureincludes: a voice input interface configured to recognize a voice input;a script input interface configured to recognize a script input; and aprocessor configured to determine whether a voice activation conditionindicating whether a user's voice input is started is satisfied,activate the voice input interface based on a result of the determining,and simultaneously provide the activated voice input interface and thescript input interface.

Hereinafter, embodiments of the present disclosure will be described indetail with reference to the accompanying drawings so that the presentdisclosure may be easily carried out by those of ordinary skill in theart. However, the present disclosure may be embodied in many differentforms and is not limited to the embodiments described herein. Also, inorder to clearly describe the present disclosure, parts having norelation to the description are omitted, and like reference numerals areassigned to like elements throughout the specification.

Some embodiments of the present disclosure may be represented byfunctional block configurations and various processing operations. Allor part of such functional blocks may be implemented by any number ofhardware and/or software elements configured to perform particularfunctions. For example, the functional blocks of the present disclosuremay be implemented as one or more microprocessors or may be implementedas circuit configurations for certain functions. Also, for example, thefunctional blocks of the present disclosure may be implemented withvarious programming or scripting languages. The functional blocks may beimplemented as algorithms that are executed on one or more processors.Also, the present disclosure may employ any conventional techniques forelectronic environment configuration, signal processing, and/or dataprocessing.

Also, the connecting lines or connecting members illustrated in thedrawings are intended to represent exemplary functional connectionsand/or physical or logical connections between elements. In a realdevice, connections between elements may be represented by variousfunctional connections, physical connections, or circuit connectionsthat are replaceable or added.

Also, the terms “-er/or” and “module” as used herein represent a unitfor processing at least one function or operation, which may beimplemented by hardware, software, or a combination thereof. The terms“-er/or” and “module” may be implemented by a program that is stored inan addressable storage medium and is executable by a processor.

For example, the terms “-er/or” and “module” may refer to elements suchas software elements, object-oriented software elements, class elements,and task elements, and may include processes, functions, attributes,procedures, subroutines, segments of program code, drivers, firmware,microcode, circuits, data, a database, data structures, tables, arrays,and variables.

The embodiments of the present disclosure, which are described in thisspecification and drawings, are merely presented as specific examples soas to easily explain the technical contents of the present disclosureand help the understanding of the present disclosure and are notintended to limit the scope of the present disclosure. That is, it willbe obvious to those of ordinary skill in the art that othermodifications based on the technical idea of the present disclosure maybe made. Also, the respective embodiments may be operated in combinationwith each other as necessary. For example, some portions of oneembodiment of the present disclosure and some portions of anotherembodiment of the present disclosure may be combined with each other sothat the device may be operated.

FIG. 1 is a diagram of a text input device according to an embodiment.

Referring to FIG. 1, a text input device 100 includes a processor 110, ascript input interface 120, and a voice input interface 130. Theprocessor 110 controls overall operations of the text input device 100.

A user 10 may provide a voice input or a script input to the text inputdevice 100. The voice input is a user's voice including information, forexample, text information. The script input is a user's action includinginformation, for example, text information. The user's action mayinclude a contact with the device.

The text input device 100 may receive the script input by using thescript input interface 120 and may receive the voice input by using thevoice input interface 130. The script input interface 120 may include akeyboard user interface (UI) on which characters, numbers, and symbolsare displayed. For UIs included in the script input interface 120,characters may be different for each type of language. For example, a UIfor Korean input and a UI for English input may exist separately. Thetext input device 100 may simultaneously provide the script inputinterface 120 and the voice input interface 130.

The processor 110 may determine whether a voice activation condition anda voice deactivation condition are satisfied. The voice activationcondition may be a result of determining the intention of the user 10 toprovide the voice input. The voice deactivation condition may be aresult of determining the intention of the user 10 to end providing thevoice input. The processor 110 may automatically activate or deactivatethe voice input interface 130 based on whether the determined voiceactivation condition or voice deactivation condition is satisfied.

In an embodiment, the processor 110 may determine whether the voiceactivation condition is satisfied, based on sound information receivedby using a microphone. The processor 110 according to the embodiment mayactivate the microphone based on a previously determined condition. Forexample, the processor 110 may activate the microphone when the scriptinput is detected. The processor 110 may determine whether user voiceinformation is included in the sound information received by using themicrophone, and may determine whether the voice activation condition issatisfied, based on a result of the determining.

In an embodiment, the processor 110 may determine whether the voiceactivation condition is satisfied, based on a slope of the text inputdevice 100. The slope of the text input device 100 may be measured byusing an accelerometer or a gyroscope. The processor 110 may determinewhether the voice activation condition is satisfied by comparing theslope of the text input device 100 with a threshold value.Alternatively, the processor 110 may determine whether the voiceactivation condition is satisfied, based on whether the slope of thetext input device 100 is a slope suitable for user face recognition.However, the operation of determining the voice activation conditionbased on the slope of the text input device 100 is not limited to theabove-described embodiments.

In an embodiment, the processor 110 may determine whether the voiceactivation condition is satisfied, based on a distance from the user'sface to the text input device 100. The processor 110 according to theembodiment may determine the distance from the user's face to the textinput device 100 based on image information received by using a camera.The processor 110 may determine whether the voice activation conditionis satisfied by comparing the aforementioned distance with a thresholdvalue. Alternatively, the processor 110 may determine whether the voiceactivation condition is satisfied, based on whether the aforementioneddistance is a distance suitable for user face recognition. However, theoperation, performed by the processor 110, of determining the voiceactivation condition based on the distance is not limited to theabove-described embodiments.

In an embodiment, the processor 110 may determine whether the voiceactivation condition is satisfied, based on a determination of a user'slip movement. The processor 110 according to the embodiment may activatea camera on the user side, for example, a front camera, based on apreviously determined condition. For example, the processor 110 mayactivate the camera when the script input is detected. The processor 110may determine the user's lip movement based on image informationreceived by using a camera. For example, the processor 110 may determinewhether the voice activation condition is satisfied, based on whetherthe user's lip movement included in the received image informationcorresponds to the voice input.

In an embodiment, the processor 110 may determine whether the voiceactivation condition is satisfied, based on context information. Thecontext information may include at least one of surrounding environmentinformation of the text input device 100, state information of the textinput device 100, user state information, user's usage historyinformation of the text input device 100, and user schedule information,but the present disclosure is not limited thereto.

The surrounding environment information of the text input device 100refers to environment information within a certain radius from the textinput device 100, and may include weather information, temperatureinformation, humidity information, illumination information, noiseinformation, sound information, time information, and the like. Forexample, when the surrounding environment is a crowded area, the textinput device 100 may determine that the voice activation condition isnot satisfied. Alternatively, when a current time is not a late time,the text input device 100 may satisfy the voice activation conditionwith a higher frequency. However, the surrounding environmentinformation included in the context information is not limited thereto.

The state information of the text input device 100 may include modeinformation of the text input device 100 (e.g., a sound mode, avibration mode, a silent mode, a power saving mode, a blocking mode, amulti-window mode, an automatic rotation mode, etc.), locationinformation of the text input device 100, time information,communication module activation information (e.g., Wi-Fi ON/BluetoothOFF/GPS ON/NFC ON, etc.), network connection state information of thetext input device 100, information about an application executed in thetext input device 100 (e.g., application identification information,application type, application use time, application use cycle, etc.),and the like. For example, the text input device 100 may determinewhether the voice activation condition is satisfied, based on a posture,an angle, a relative position, and the like of the text input device 100when the user starts a conversation. However, the state information ofthe text input device included in the context information is not limitedthereto.

The user state information is information about a user'scharacteristics, movement, and life pattern, and the like, and mayinclude information about a user's gender, a user's walking state,exercising state, driving state, and sleeping state, a user's moodstate, and the like. For example, when a gesture that the user usesfrequently during a voice input is recognized, the text input device 100may satisfy the voice activation condition with a higher frequency.However, the user state information included in the context informationis not limited thereto.

The user's usage history information of the text input device 100 isinformation about the history of the user using the text input device100, and may include execution history of the application, the historyof functions executed in the application, a user's call history, auser's text history, and the frequency of words included in the textinformation. For example, the text input device may satisfy the voiceactivation condition with a higher frequency during a text conversationwith a specific user. However, the usage history information included inthe context information is not limited thereto.

In determining whether the voice activation condition is satisfied,which context information is to be used may be determined according tolearning based on a preset criterion. For example, supervised learningthat uses a certain voice input and certain context information as inputvalues and unsupervised learning that discovers a pattern of a voiceactivation condition by self-learning a type of context informationnecessary for determining a voice activation condition without separatesupervision may be used to determine the voice activation condition.Also, for example, reinforcement learning that uses a feedback on aresult of grasping the user's intention according to learning is correctmay be used to determine the voice activation condition.

In an embodiment, the processor 110 may determine whether the voiceactivation condition is satisfied by combining the above-describedembodiments in parallel or in combination.

The processor 110 may activate the voice input interface 130 forrecognizing the voice input, based on a result of determining whetherthe voice activation condition is satisfied. The processor 110 maysimultaneously provide the voice input interface 130 and the scriptinput interface 120.

The processor 110 generates voice text information based on the voiceinput recognized by using the voice input interface 130. Also, theprocessor 110 generates script text information based on the scriptinput recognized by using the script input interface 120. The processor110 may generate text information by combining the voice textinformation with the script text information.

The processor 110 may control a display (not illustrated) of the textinput device 100 such that the generated text information is displayedthereon. In order to display the text information, the processor 110 maydisplay the script text information and the voice text information inthe order of generation.

The processor 110 may determine whether the voice deactivation conditionis satisfied. The voice deactivation condition may be a result ofdetermining the intention of the user 10 to end providing the voiceinput.

In an embodiment, the processor 110 may determine whether the voicedeactivation condition is satisfied, based on context information.Alternatively, the processor 110 may determine whether the voicedeactivation condition is satisfied, based on sound information receivedby using a microphone, a slope of the text input device 100, a distancefrom a user's face, and a determination of a user's lip movement. Forexample, the processor 110 may determine whether the voice deactivationcondition is satisfied, based on whether the slope of the text inputdevice 100 is a slope unsuitable for user face recognition. Also, theprocessor 110 may determine the voice deactivation condition based on anelement used to determine whether the voice activation condition issatisfied. However, the operation, performed by the processor 110, ofdetermining the voice deactivation condition is not limited to theabove-described embodiments.

The processor 110 may deactivate the voice input interface 130 based ona result of determining whether the voice deactivation condition issatisfied. The text input device 100 may simultaneously provide thescript input interface 120 and the voice input interface 130 and controlwhether to activate the voice input interface 130 automatically oraccording to the user's selection, based on the voice activationcondition or the voice deactivation condition. Thus, the text inputoperation may be efficiently performed.

Examples of the text input device 100 may include smartphones, tabletpersonal computers (PCs), PCs, smart televisions (TVs), mobile phones,personal digital assistants (PDAs), laptop computers, media players,micro servers, global positioning system (GPS) devices, e-bookterminals, digital broadcasting terminals, navigations, kiosk, MP3players, digital cameras, home appliances, and other mobile ornon-mobile computing devices, but the present disclosure is not limitedthereto. Also, examples of the text input device 100 may includewearable devices, such as watches, glasses, hair bands, or rings, whichhave a communication function and a data processing function. However,the present disclosure is not limited thereto, and the text input device100 may include any types of devices capable of receiving a user inputand providing text based thereon.

Also, the text input device 100 may communicate with a server and otherdevices (not illustrated) via a certain network in order to use variouspieces of context information. In this case, the network may include alocal area network (LAN), a wide area network (WAN), a value addednetwork (VAN), a mobile radio communication network, a satellitecommunication network, or any combinations thereof. The network may be acomprehensive data communication network that enables networkconfiguration entities to efficiently communicate with each other andmay include a wired Internet, a wireless Internet, and a mobile wirelesscommunication network. Examples of the wireless communication mayinclude wireless LAN (Wi-Fi), Bluetooth, Bluetooth Low Energy, Zigbee,Wi-Fi Direct (WFD), ultra wideband (UWB), Infrared Data Association(IrDA), and Near Field Communication (NFC), but the present disclosureis not limited thereto.

FIG. 2 is a diagram of a text input device according to an embodiment.

Referring to FIG. 2, a text input device 200 includes a processor 210, ascript input interface 220, a voice input interface 230, and a sensor240.

The processor 210 controls overall operations of the text input device200.

The text input device 200 may receive a script input by using the scriptinput interface 220 and may receive a voice input by using the voiceinput interface 230. The text input device 200 may simultaneouslyprovide the script input interface 220 and the voice input interface230.

The sensor 240 includes at least one sensor that receives informationfrom the outside. In an embodiment, the sensor 240 may include amicrophone that receives sound information. In an embodiment, the sensor240 may include an accelerometer or a gyroscope that detects movement ofthe text input device 200. In an embodiment, the sensor 240 may includea camera that receives image information. In an embodiment, the sensor240 may include a temperature sensor or a humidity sensor that receivessurrounding environment information. However, this is an example, andthe type of sensor included in the sensor 240 is not limited thereto.The sensor 240 may provide the received information to the processor210. The processor 210 may determine whether a voice activationcondition is satisfied, based on the provided information.

FIG. 3 is a diagram of a text input device according to an embodiment.

Referring to FIG. 3, a text input device 300 includes a processor 310, ascript input interface 320, a voice input interface 330, and anauxiliary voice input interface 340.

A user 30 may provide a voice input or a script input to the text inputdevice 300. The voice input is a user's voice including information, forexample, text information. The script input is a user's action includinginformation, for example, text information. The user's action mayinclude a contact with the device.

The processor 310 may control the script input interface 320 and thevoice input interface 330 such that the script input is input by usingthe script input interface 320 and the voice input is input by using thevoice input interface 330.

The processor 310 may determine whether a voice activation condition anda voice deactivation condition are satisfied. The processor 310 mayautomatically activate or deactivate the voice input interface 330 basedon a result of determining whether the voice activation condition or thevoice deactivation condition is satisfied.

The processor 310 may generate recommended voice text information fromthe voice input recognized by using the voice input interface 330, basedon context information. The processor 310 may provide the recommendedvoice text information to a user by using the auxiliary voice inputinterface 340. The recommended voice text information is textinformation expected to be input by the user, based on the voice inputand the context information.

The context information may include at least one of surroundingenvironment information of the text input device 300, state informationof the text input device 300, user state information, user's usagehistory information of the text input device 300, and user scheduleinformation, but the present disclosure is not limited thereto.

In generating the recommended voice text information, which contextinformation is to be used may be determined according to learning basedon a preset criterion. For example, supervised learning that uses acertain voice input and certain context information as input values andunsupervised learning that discovers a pattern of recommended voice textinformation by self-learning a type of context information necessary forthe recommended voice text information without separate supervision maybe used to generate the recommended voice text information. Also, forexample, reinforcement learning that uses a feedback on whether a resultof grasping the user's intention according to learning is correct may beused to generate the recommended voice text information.

The processor 310 may receive a user input of selecting one of pieces ofthe provided recommended voice text information as voice textinformation by using the auxiliary voice input interface 340. Also, theprocessor 310 may generate script text information based on the scriptinput recognized by using the script input interface 320. The processor310 may generate text information by combining the selected voice textinformation with the script text information.

The processor 310 may control a display (not illustrated) such that thegenerated text information is displayed thereon. In order to display thetext information, the processor 310 may display the script textinformation and the voice text information in the order of generation.

FIG. 4A is a diagram for describing a method, performed by a text inputdevice 400 a, of performing a text input operation, according to someembodiments.

Referring to FIG. 4A, the text input device 400 a includes a scriptinput interface 420 a and a voice input interface (not illustrated). Auser 40 may provide a voice input or a script input to the text inputdevice 400 a.

The text input device 400 a may receive the script input by using thescript input interface 420 a and may receive the voice input by usingthe voice input interface. The text input device 400 a may generatevoice text information based on the voice input recognized by using thevoice input interface and may generate script text information based onthe script input recognized by using the script input interface 420 a.The text input device 400 a may generate text information by combiningthe voice text information with the script text information.

In FIG. 4A, the script input interface 420 a is displayed in the form ofa QWERTY keyboard UI, but this is an example. The form of the scriptinput interface 420 a according to the present embodiment is not limitedto the aforementioned keyboard UI. Also, although the voice inputinterface is not illustrated in FIG. 4A, this is an example. The scriptinput interface 420 a may be displayed on the text input device 400 a inthe form of a UI.

The text input device 400 a may provide the script input interface 420 awhen a text input request is detected. For example, the text inputdevice 400 a may provide the script input interface 420 a when a userinput of requesting a UI for text input is detected. Alternatively, thetext input device 400 a may provide the script input interface inresponse to a request from an application executed in the text inputdevice 400 a.

When the text input device 400 a provides the script input interface 420a, the text input device 400 a may set a text display area 401 a that isa preset area for displaying text information. For example, when asearch application is executed, a search window may be included in thetext display area 401 a.

As described above, the text input device 400 a may generate script textinformation based on the script input recognized by using the scriptinput interface 320. The script input may include a text of at least oneof characters, numbers, and symbols. The text input device 400 a maydisplay the generated script text information on the text display area401 a.

The text input device 400 a may determine whether a voice activationcondition, which is a result of determining the intention of the user toprovide the voice input, is satisfied. In an embodiment, the text inputdevice 400 a may determine whether the voice activation condition issatisfied, based on a combination of the presence or absence of scriptinput, sound information received by using a microphone, a slope of thetext input device 400 a, a distance from a user's face, and a user's lipmovement. However, the method of determining the voice activationcondition is not limited to the above-described embodiments.

The text input device 400 a may provide the voice input interface forrecognizing the voice input together with the script input interface 420a, based on the determined voice activation condition.

In an embodiment, the text input device 400 a may display a state of thevoice input interface, that is, whether activated or deactivated, whilethe voice input interface is provided. For example, the text inputdevice 400 a may change a color of a voice input activation icon 402 adisplayed on the text input device 400 a, based on the state of thevoice input interface. The voice input activation icon 402 a may belocated on the script input interface 420 a. However, the method ofdisplaying the state of the voice input interface while the voice inputinterface is provided is not limited to the above-described embodiments.

As described above, the text input device 400 a may generate voice textinformation based on the voice input recognized by using the voice inputinterface. The text input device 400 a may display the generated voicetext information on the text display area 401 a.

In an embodiment, when a voice input interface is provided together withthe script input interface, the text input device 400 a may display, onthe text display area 401 a, the voice text information or the scripttext information generated based on the voice input or the script input.

In an embodiment, the text input device 400 a may display the voice textinformation or the script text information on the text display area 401a in the order of input. When the inputs are performed at the same time,the text input device 400 a may determine a display order of voice textinformation or script text information according to a pre-designatedpriority flag. The priority flag according to the embodiment may bedesignated by the user or may be pre-designated and stored in a memory(not illustrated).

FIG. 4B is a diagram for describing a method, performed by a text inputdevice 400 b, of performing a text input operation, according to someembodiments.

Referring to FIG. 4B, the text input device 400 b includes a scriptinput interface 420 b, a voice input interface (not illustrated), and anauxiliary voice input interface 440 b. A user 40 may provide a voiceinput or a script input to the text input device 400 b.

The text input device 400 b may receive the script input by using thescript input interface 420 b and may receive the voice input by usingthe voice input interface.

In FIG. 4B, the script input interface 420 b is displayed in the form ofa QWERTY keyboard UI, but this is an example. The form of the scriptinput interface 420 b according to the present embodiment is not limitedto the aforementioned keyboard UI. Also, although the voice inputinterface is not illustrated in FIG. 4B, this is an example. The voiceinput interface may be displayed on the text input device 400 b in theform of a UI.

The text input device 400 b provides the script input interface 420 bwhen a text input request is detected. When the text input device 400 bprovides the script input interface 420 b, the text input device 400 bmay set a text display area 401 b that is a preset area for displayingtext information.

The text input device 400 b may generate script text information basedon the script input recognized by using the script input interface 420b. The text input device 400 b may display the generated script textinformation on the text display area 401 b.

The text input device 400 b may determine whether a voice activationcondition, which is a result of determining the intention of the user toprovide the voice input, is satisfied. In an embodiment, the text inputdevice 400 b may determine whether the voice activation condition issatisfied, based on a combination of the presence or absence of scriptinput, sound information received by using a microphone, a slope of thetext input device 400 b, a distance from a user's face, and a user's lipmovement. However, the method of determining the voice activationcondition is not limited to the above-described embodiments.

The text input device 400 b may provide the voice input interface forrecognizing the voice input together with the script input interface 420b, based on the determined voice activation condition.

In an embodiment, the text input device 400 b may display a state of thevoice input interface, that is, whether activated or deactivated, whilethe voice input interface is provided. For example, the text inputdevice 400 b may change a color of a voice input activation icon 402 bdisplayed on the text input device 400 b, based on the state of thevoice input interface. However, the method of displaying the state ofthe voice input interface while the voice input interface is provided isnot limited to the above-described embodiments.

The text input device 400 b may generate recommended voice textinformation from the voice input received by using the voice inputinterface, based on context information. The recommended voice textinformation is text information expected to be input by the user, basedon the voice input and the context information. For example, when thetext input device 400 b determines that text information “text” isincluded in the voice input, the text input device 400 b may generate“texture,” “text,” and “texted” as the recommended voice textinformation.

The text input device 400 b may provide the recommended voice textinformation to the user by using the auxiliary voice input interface 440b. Also, the text input device 400 b may receive a user input ofselecting one of pieces of the provided recommended voice textinformation as voice text information by using the auxiliary voice inputinterface 440 b.

The text input device 400 b may display the selected voice textinformation on the text display area 401 b.

More specifically, when a voice input interface is provided togetherwith the script input interface, the text input device 400 b maydisplay, on the text display area 401 b, the voice text information orthe script text information generated based on the voice input or thescript input.

In an embodiment, the text input device 400 b may display the voice textinformation or the script text information on the text display area 401b in the order of input. When the inputs are performed at the same time,the text input device 400 b may determine a display order of voice textinformation or script text information according to a pre-designatedpriority flag. The priority flag according to the embodiment may bedesignated by the user or may be pre-designated and stored in a memory(not illustrated).

FIG. 5 is a diagram for describing a method, performed by a text inputdevice, of performing a text input operation based on contextinformation, according to some embodiments.

Referring to FIG. 5, text input devices 500 a and 500 b include scriptinput interfaces 520 a and 520 b, voice input interfaces (notillustrated), and auxiliary voice input interfaces 540 a and 540 b,respectively. A user may provide voice inputs or script inputs to thetext input devices 500 a and 500 b.

The text input devices 500 a and 500 b may receive the script inputs byusing the script input interfaces 520 a and 520 b, respectively, and mayreceive the voice inputs by using the voice input interfaces,respectively. When the text input devices 500 a and 500 b provide thescript input interfaces 520 a and 520 b, respectively, the text inputdevices 500 a and 500 b may set text display areas 501 a and 501 b thatare a preset area for displaying text information, respectively.

The text input devices 500 a and 500 b may determine whether a voiceactivation condition, which is a result of determining the intention ofthe user to provide the voice input, is satisfied. In an embodiment, thetext input devices 500 a and 500 b may determine whether the voiceactivation condition is satisfied, based on a combination of thepresence or absence of script input, sound information received by usinga microphone, slopes of the text input devices 500 a and 500 b, adistance from a user's face, and a user's lip movement. However, themethod of determining the voice activation condition is not limited tothe above-described embodiments.

The text input devices 500 a and 500 b may provide the voice inputinterfaces for recognizing the voice inputs together with the scriptinput interfaces 520 a and 520 b, based on the determined voiceactivation condition, respectively.

The text input devices 500 a and 500 b may generate recommended voicetext information from the voice inputs received by using the voice inputinterfaces, based on context information, respectively. The recommendedvoice text information is text information expected to be input by theuser, based on the voice input and the context information.

In an embodiment, the context information includes gender information ofthe user. In an embodiment, the gender information of the user may bedetermined from the user's voice input, or may be previously stored orlearned and determined.

Referring to FIG. 5, when the gender of the user included in the contextinformation is female, the text input device 500 a may generate“fiancee,” “husband,” and “boyfriend” as recommended voice textinformation. When the gender of the user included in the contextinformation is male, the text input device 500 b may generate “fiancee,”“wife,” and “girlfriend” as recommended voice text information. Eachword disclosed in FIG. 5 is an example, and the methods, performed bythe text input devices 500 a and 500 b, of generating the recommendedvoice text information based on the gender information are not limitedthereto.

FIG. 6 is a flowchart for describing a text input method according to anembodiment.

In operation S610, a text input device provides a script input interfacefor receiving a script input. The script input is a user's actionincluding information, for example, text information. The user's actionmay include a contact with the device. The script input interface mayinclude a keyboard UI on which characters, numbers, and symbols aredisplayed. In an embodiment, the text input device may provide thescript input interface when a user input of requesting an interface fortext input is detected.

In operation S620, the text input device determines whether a voiceactivation condition is satisfied. The voice activation condition may bea result of determining the intention of the user to provide the voiceinput. In an embodiment, the text input device may determine whether thevoice activation condition is satisfied, based on a combination of thepresence or absence of script input, sound information received by usinga microphone, a slope of the text input device, a distance from a user'sface, and a user's lip movement. Alternatively, when the script inputinterface is provided, the text input device may automatically determinethat the voice activation condition is satisfied. However, the method ofdetermining the voice activation condition is not limited to theabove-described embodiments.

In operation S630, the text input device provides a voice inputinterface for recognizing voice input together with the script inputinterface, based on the determined voice activation condition. Forexample, when the user's voice is included in the sound informationreceived by using the microphone and the voice activation condition issatisfied, the text input device may provide the voice input interfacetogether with the script input interface when the user's voice ispresent. Alternatively, when the script input interface is provided andthe voice activation condition is satisfied, the text input device mayautomatically provide the script input interface and the voice inputinterface at the same time at any time. That is, when the voiceactivation condition is satisfied, the text input device may activatethe voice input interface to provide the user with a text input usingboth the voice input and the script input.

FIG. 7 is a flowchart for describing a method of determining a voiceactivation condition, according to an embodiment.

In operation S710, a text input device activates a microphone. The textinput device may activate the microphone when a preset condition issatisfied. For example, the text input device may activate themicrophone when a script input is present. In an embodiment, when ascript input that is input via the script input interface within acertain time is present, the text input device may determine that thescript input is present.

In operation S720, the text input device receives sound information byusing the microphone. The sound information is auditory informationaround the text input device, which is received by using the microphone.

In operation S730, the text input device determines whether a user'svoice is included in the received sound information. The method,performed by the text input device, of determining whether the user'svoice is included may be performed based on various algorithms. Also,the text input device may perform noise cancellation and various signalprocessing operations in order to determine whether the user's voice isincluded.

In operation S740, the text input device determines whether the voiceactivation condition is satisfied, based on whether the user's voice isincluded. When the user's voice is recognized, the text input device maydetermine that the voice activation condition has been satisfied, andprovide the user with a text input via voice.

FIG. 8 is a flowchart for describing a method of determining a voiceactivation condition, according to an embodiment.

In operation S810, the text input device measures a slope of the textinput device. In an embodiment, the text input device may measure theslope of the text input device by using an accelerometer or a gyroscope.In an embodiment, the text input device may perform a slope measuringoperation when a preset condition is satisfied. For example, the textinput device may measure the slope when a script input is present. In anembodiment, when a script input that is input via the script inputinterface within a certain time is present, the text input device maydetermine that the script input is present.

In operation S820, the text input device determines whether the voiceactivation condition is satisfied, based on the measured slope. In anembodiment, the text input device may determine whether the voiceactivation condition is satisfied by comparing the measured slope with athreshold value. Alternatively, the text input device may determinewhether the voice activation condition is satisfied, based on whetherthe measured slope is a slope suitable for user face recognition.However, the operation of determining the voice activation conditionbased on the slope of the text input device is not limited to theabove-described embodiments.

FIG. 9 is a flowchart for describing a method of determining a voiceactivation condition, according to an embodiment.

In operation S910, the text input device measures a distance from auser's face to the text input device. In an embodiment, the text inputdevice may measure the aforementioned distance by using a camera on theuser side, for example, a front camera. In an embodiment, the text inputdevice may activate the camera when a preset condition is satisfied. Forexample, the text input device may activate the camera when a scriptinput is present. Alternatively, the text input device may activate thecamera when the slope of the text input device satisfies a certaincondition. However, the camera activating operation and the distancemeasuring operation of the text input device are not limited to theabove-described embodiments.

In operation S920, the text input device determines whether the voiceactivation condition is satisfied, based on the measured distance. In anembodiment, the text input device may determine whether the voiceactivation condition is satisfied by comparing the measured distancewith a threshold value. Alternatively, the text input device maydetermine whether the voice activation condition is satisfied, based onwhether the measured distance is a distance suitable for user facerecognition. However, the operation, performed by the text input device,of determining the voice activation condition based on the distance isnot limited to the above-described embodiments.

FIG. 10 is a flowchart for describing a method of determining a voiceactivation condition, according to an embodiment.

In operation S1010, the text input device activates a camera to receiveimage information. In an embodiment, the text input device may activatea camera on the user side, for example, a front camera. In anembodiment, the text input device may activate the camera when a presetcondition is satisfied. For example, the text input device may activatethe camera when a script input is present. Alternatively, the text inputdevice may activate the camera when the slope of the text input devicesatisfies a certain condition. However, the camera activating operationand the distance measuring operation of the text input device are notlimited to the above-described embodiments.

In operation S1020, the text input device determines a user's lipmovement based on the received image information. In an embodiment, thetext input device may recognize a user's face from the imageinformation.

In operation S1030, the text input device determines whether a voiceactivation condition is satisfied, based on the determined lip movement.In an embodiment, the text input device may determine whether the voiceactivation condition is satisfied, based on whether the lip movement ispresent. Alternatively, the text input device may determine whether thevoice activation condition is satisfied, based on whether the determinedlip movement includes text information. However, the operation,performed by the text input device, of determining the voice activationcondition based on the lip movement is not limited to theabove-described embodiments.

FIG. 11 is a flowchart for describing a method of determining a voiceactivation condition, according to an embodiment.

In operation S1110, a text input device measures a slope of the textinput device. In an embodiment, the text input device may measure theslope of the text input device by using an accelerometer or a gyroscope.For example, the text input device may perform a slope measuringoperation when a preset condition is satisfied.

In operation S1120, the text input device activates a camera to receiveimage information. In an embodiment, the text input device may receivethe image information by using a camera on the user side, for example, afront camera. In an embodiment, the text input device may activate thecamera when a preset condition is satisfied. For example, the text inputdevice may activate the camera when a script input is present.Alternatively, the text input device may activate the camera when theslope of the text input device satisfies a certain condition.

In operation S1130, the text input device measures a distance from auser's face to the text input device. In an embodiment, the text inputdevice may measure the aforementioned distance based on the imageinformation received by using the camera.

In operation S1140, the text input device activates a microphone toreceive sound information. The text input device may activate themicrophone when a preset condition is satisfied. For example, the textinput device may activate the microphone together when the camera isactivated. Alternatively, the text input device may activate themicrophone when a distance between the text input device and the user'sface satisfies a certain condition.

In operation S1150, the text input device determines whether a voiceactivation condition is satisfied. For example, when the user's voice isrecognized from the sound information received by using the microphone,the text input device may determine that the voice activation conditionhas been satisfied. Alternatively, when the user's lip movement isrecognized from the image information received by using the camera, thetext input device may determine that the voice activation condition hasbeen satisfied. Alternatively, when the user's voice is recognized fromthe sound information and the user's lip movement is recognized from theimage information received by using the camera, the text input devicemay determine that the voice activation condition has been satisfied.The text input device may determine whether the voice activationcondition is satisfied by using sensing operations of various sensors,and provide the user with a text input via voice.

FIG. 12 is a flowchart for describing a text input method according toan embodiment.

In operation S1210, a text input device provides a script inputinterface for recognizing a script input. In an embodiment, the textinput device may provide the script input interface when a user input ofrequesting an interface for text input is detected.

In operation S1220, the text input device determines whether a voiceactivation condition is satisfied. In an embodiment, the text inputdevice may determine whether the voice activation condition issatisfied, based on a combination of the presence or absence of scriptinput, sound information received by using a microphone, a slope of thetext input device, a distance from a user's face, and a user's lipmovement.

In operation S1230, the text input device simultaneously provides avoice input interface for recognizing voice input and a script inputinterface, based on a result of determining whether the voice activationcondition is satisfied.

In operation S1240, the text input device determines whether a voicedeactivation condition is satisfied. The voice deactivation condition isa result of determining the intention of the user to end providing thevoice input. In an embodiment, the text input device may determinewhether the voice deactivation condition is satisfied, based on soundinformation received by using a microphone, a slope of the text inputdevice, a distance from a user's face, a user's lip movement, and thelike. For example, the text input device may determine whether the voicedeactivation condition is satisfied, based on whether the slope of thetext input device is a slope unsuitable for user face recognition. Also,the text input device may determine the voice deactivation conditionbased on an element used to determine whether the voice activationcondition is satisfied.

In operation S1250, the text input device deactivates the voice inputinterface, based on whether the voice deactivation condition issatisfied. The text input device may simultaneously provide the scriptinput interface and the voice input interface and control whether toactivate the voice input interface automatically or according to theuser's selection, based on the voice activation condition or the voicedeactivation condition. Thus, the text input operation may beefficiently performed.

FIGS. 13 and 14 are block diagrams of a first device 1000 according tosome embodiments.

As illustrated in FIG. 13, the first device 1000 according to someembodiments may include a user inputter 1100, an outputter 1200, aprocessor 1300, and a communicator 1500. However, all elementsillustrated in FIG. 13 are not essential to the first device 1000. Thefirst device 1000 may include more elements than those illustrated inFIG. 13 or may include fewer elements than those illustrated in FIG. 13.

For example, as illustrated in FIG. 14, the first device 1000 accordingto some embodiments may further include, in addition to the userinputter 1100, the outputter 1200, the processor 1300, and thecommunicator 1500, a sensor 1400, an audio/video (A/V) inputter 1600,and a memory 1700.

The user inputter 1100 is a device that allows a user to input data forcontrolling the first device 1000. For example, the user inputter 1100may include a key pad, a dome switch, a touch pad (e.g., a touch-typecapacitive touch pad, a pressure-type resistive touch pad, an infraredbeam sensing type touch pad, a surface acoustic wave type touch pad, anintegral strain gauge type touch pad, a piezo effect type touch pad,etc.), a jog wheel, and a jog switch, but the present disclosure is notlimited thereto.

The user inputter 1100 may include a script input interface and a voiceinput interface. The user inputter 1100 may receive a voice input or ascript input. Also, the user inputter 1100 may receive a user input ofselecting one of pieces of recommended voice text information as voicetext information.

The outputter 1200 may output an audio signal, a video signal, or avibration signal. The outputter 1200 may include a display 1210, anaudio outputter 1220, and a vibration motor 1230.

The display 1210 displays information processed by the first device1000. For example, the display 1210 may display the script inputinterface for receiving the script input or a voice input activationicon indicating whether the voice input interface is activated. Also,the display 1210 may display text information and a text display areathat is a preset area for displaying the text information.

When the display 1210 and the touch pad form a layered structure toconstitute a touch screen, the display 1210 may also be used as an inputdevice as well as an output device. The display 1210 may include atleast one selected from liquid crystal display (LCD), thin-filmtransistor-liquid crystal display (TFT-LCD), organic light-emittingdiode (OLED), flexible display, three-dimensional (3D) display, andelectrophoretic display. According to the implementation type of thefirst device 1000, the first device 1000 may include two or moredisplays 1210. The two or more displays 1210 may be arranged to faceeach other by using a hinge.

The audio outputter 1220 outputs audio data that is received from thecommunicator 1500 or is stored in the memory 1700. Also, the audiooutputter 1220 outputs an audio signal associated with the functionperformed by the first device 1000 (e.g., a call signal reception sound,a message reception sound, a notification sound, etc.). The audiooutputter 1220 may include a speaker, a buzzer, and the like.

The vibration motor 1230 may output a vibration signal. For example, thevibration motor 1230 may output a vibration signal corresponding to theoutput of audio data or video data (e.g., a call signal reception sound,a message reception sound, etc.). Also, the vibration motor 1230 mayoutput a vibration signal when a touch is input to a touch screen.

The processor 1300 controls overall operations of the first device 1000.For example, the processor 1300 may execute programs stored in thememory 1700 to control the user inputter 1100, the outputter 1200, thesensor 1400, the communicator 1500, and the A/V inputter 1600.

Specifically, the processor 1300 may control the user inputter 1100 suchthat a script input is input by using a script input interface and avoice input is input by using a voice input interface.

The processor 1300 may determine whether a voice activation conditionand a voice deactivation condition are satisfied. In an embodiment, theprocessor 1300 may determine whether the voice activation condition andthe voice deactivation condition are satisfied, based on a combinationof the presence or absence of script input, sound information receivedby using a microphone, a slope of the first device 1000, a distance froma user's face, and a user's lip movement. However, the method ofdetermining the voice activation condition is not limited to theabove-described embodiments.

The processor 1300 may automatically activate or deactivate the voiceinput interface included in the user inputter 1100, based on thedetermined voice activation condition or the determined voicedeactivation condition.

Also, the processor 1300 may generate recommended voice text informationfrom the voice input recognized by using the user inputter 1100, basedon context information. The processor 1300 may provide the generatedrecommended voice text information to the user.

The processor 1300 may control the user inputter 1100 to receive a userinput of selecting one of pieces of the provided recommended voice textinformation as voice text information by using the auxiliary voice inputinterface included in the user inputter 1100. Also, the processor 1300may generate script text information based on the script inputrecognized by using the script input interface. The processor 1300 maygenerate text information by combining the selected voice textinformation with the script text information.

The processor 1300 may control the display 1210 such that the generatedtext information is displayed thereon. In order to display the textinformation, the processor 1300 may display the script text informationand the voice text information in the order of generation.

The sensor 1400 may sense a state of the first device 1000 or a statearound the first device 1000 and transmit sensed information to theprocessor 1300.

The sensor 1400 may include at least one of a geomagnetic sensor 1410,an acceleration sensor 1420, a temperature/humidity sensor 1430, aninfrared sensor 1440, a gyroscope sensor 1450, a position sensor (e.g.,GPS) 1460, a barometric sensor 1470, a proximity sensor 1480, and an RGBsensor (illuminance sensor) 1490, but the present disclosure is notlimited thereto. Because the functions of the respective sensors may beintuitively inferred from their names, detailed descriptions thereofwill be omitted.

The communicator 1500 may include one or more elements for communicationbetween the first device 1000 and a head mounted display (HMD) device orcommunication between the first device 1000 and a server. For example,the communicator 1500 may include a short-range wireless communicator1510, a mobile communicator 1520, and a broadcasting receiver 1530.

The short-range wireless communicator 1510 may include a Bluetoothcommunicator, a Bluetooth Low Energy (BLE) communicator, a near fieldcommunicator, a wireless local area network (WLAN) (Wi-Fi) communicator,a Zigbee communicator, an infrared data association (IrDA) communicator,a Wi-Fi direct (WFD) communicator, an ultra-wideband (UWB) communicator,an Ant+ communicator, and the like, but the present disclosure is notlimited thereto.

The mobile communicator 1520 may transmit and receive a wireless signalwith at least one of a base station, an external terminal, or a serveron a mobile communication network. Examples of the wireless signal mayinclude a voice call signal, a video call signal, and various types ofdata according to transmission or reception of text/multimedia messages.

The broadcasting receiver 1530 may receive a broadcasting signal and/orbroadcasting-related information from the outside via a broadcastingchannel. The broadcasting channel may include a satellite channel, aground wave channel, or the like. The first device 1000 may not includethe broadcasting receiver 1530 according to an implementation example.

Also, the communicator 1500 may transmit or receive information forusing context information with an HMD device, a server, and a peripheraldevice.

The A/V inputter 1600 may be provided for input of an audio signal or avideo signal. For the input of the audio signal or the video signal, theA/V inputter 1600 may include a camera 1610, a microphone 1620, and thelike. The camera 1610 may obtain an image frame, such as a still imageor a moving image, through an image sensor in a video call mode or animage capture mode. An image captured by the image sensor may beprocessed by the processor 1300 or a separate image processor (notillustrated).

The image frame processed by the camera 1610 may be stored in the memory1700 or may be transmitted to the outside through the communicator 1500.Alternatively, the image frame may be used for the operation, performedby the processor 1300, of determining the voice activation condition andthe voice deactivation condition. The camera 1610 may include two ormore cameras according to a configuration of a terminal.

The microphone 1620 may receive an external audio signal and process theexternal audio signal into electrical voice data. For example, themicrophone 1620 may receive an audio signal from an external device or aperson. The microphone 1620 may use various noise cancellationalgorithms for cancelling noise generated in a process of receiving anexternal audio signal. The audio signal received through the microphone1620 may be used for the operation, performed by the processor 1300, ofdetermining the voice activation condition and the voice deactivationcondition.

The memory 1700 may store programs for processing and control of theprocessor 1300 and may store data input to the first device 1000 or dataoutput from the first device 1000.

The memory 1700 may include at least one type of storage medium selectedfrom among a flash memory type, a hard disk type, a multimedia cardmicro type, a card type memory (for example, a secure digital (SD) orextreme digital (XD) memory), random access memory (RAM), static randomaccess memory (SRAM), read-only memory (ROM), electrically erasableprogrammable ROM (EEPROM), programmable ROM (PROM), magnetic memory,magnetic disk, and optical disk.

The programs stored in the memory 1700 may be classified into aplurality of modules according to functions thereof. For example, theprograms may be classified into a UI module 1710, a touch screen module1720, and a notification module 1730.

The UI module 1710 may provide a specialized UI or GUI thatinteroperates with the first device 1000 according to each application.The touch screen module 1720 may detect a user's touch gesture on atouch screen and transmit information about the touch gesture to theprocessor 1300. The touch screen module 1720 according to someembodiments may recognize and analyze a touch code. The touch screenmodule 1720 may be implemented as separate hardware including acontroller.

Various sensors may be provided inside or near the touch screen so as todetect a touch on the touch screen or hovering above the touch screen.An example of the sensor that detects the touch on the touch screen maybe a tactile sensor. The tactile sensor may detect a contact of aspecific object at or beyond a sensitivity of a person. The tactilesensor may sense various pieces of information, such as roughness of acontact surface, hardness of a contact object, or a temperature of acontact point.

Also, an example of the sensor that detects the touch on the touchscreen may be a proximity sensor.

The proximity sensor refers to a sensor that detects the presence orabsence of an object approaching a certain detection surface or anobject existing near the sensor by using an electromagnetic force orinfrared light, without mechanical contact. Examples of the proximitysensor include a transmission-type photoelectric sensor, a directreflection-type photoelectric sensor, a mirror reflection-typephotoelectric sensor, a high frequency oscillation-type proximitysensor, a capacity-type proximity sensor, a magnetic proximity sensor,and an infrared-type proximity sensor. The user's touch gesture mayinclude a tap, a touch-and-hold, a double-tap, a drag, a panning, aflick, a drag-and-drop, a swipe, and the like.

The notification module 1730 may output a signal for notifying that anevent occurs in the first device 1000. Examples of events occurring inthe first device 1000 include a call signal reception, a messagereception, a key signal input, and a schedule notification. Thenotification module 1730 may output a notification signal through thedisplay 1210 in the form of a video signal. The notification module 1730may output a notification signal through the audio outputter 1220 in theform of an audio signal. The notification module 1730 may output anotification signal through the vibration motor 1230 in the form of avibration signal.

FIG. 15 is a block diagram of the processor 1300 according to someembodiments.

Referring to FIG. 15, the processor 1300 according to some embodimentsmay include a data trainer 1310 and a data recognizer 1320.

According to an embodiment, at least part of the data trainer 1310 andat least part of the data recognizer 1320 may be implemented as asoftware module or may be manufactured in the form of a hardware chipand mounted on a text input device.

The data trainer 1310 may learn a criterion for determining a voiceactivation and deactivation conditions and generating recommended voicetext information. The data trainer 1310 may learn a criterion regardingwhich data is to be used for determining the voice activation anddeactivation conditions and generating the recommended voice textinformation. Also, the data trainer 1310 may learn a criterion regardinghow to determine the intention of the user by using data, how to obtainand provide associated information, and how to determine the voiceactivation and deactivation conditions and generate the recommendedvoice text information. The data trainer 1310 may learn a criterion fordetermining the voice activation and deactivation conditions andgenerating the recommended voice text information by obtaining data tobe used for learning and applying the obtained data to a datarecognition model to be described below.

Types of training data input to the data trainer 1310 may include voicedata, text data, and the like.

The data recognizer 1320 may determine the voice activation anddeactivation conditions or generate the recommended voice textinformation, based on the data. The data recognizer 1320 may determinethe voice activation and deactivation conditions or generate therecommended voice text information from certain data by using thelearned data recognition model. The data recognizer 1320 may obtaincertain data according to a preset criterion obtained by learning anduse the data recognition model using the obtained data as an inputvalue. Also, the data recognizer 1320 may use the data recognition modelto determine the voice activation and deactivation conditions orgenerate the recommended voice text information, based on certain data.Also, a result value output by the data recognition model using theobtained data as the input value may be used to update the datarecognition model.

At least one of the data trainer 1310 and the data recognizer 1320 maybe manufactured in the form of at least one hardware chip and mounted onthe first device 1000. For example, at least one of the data trainer1310 and the data recognizer 1320 may be manufactured in the form of adedicated hardware chip for artificial intelligence (AI), or may bemanufactured as part of an existing general-purpose processor (e.g.,central processing unit (CPU) or application processor) or a dedicatedgraphics processor (e.g., graphics processing unit (GPU)) and mounted onthe first device 1000. In this case, the dedicated hardware chip for AIis a dedicated processor specialized for probability calculation.Because the dedicated hardware chip has higher parallel processingperformance than that of the existing general-purpose processor, it ispossible to quickly process computational operations in an AI field suchas machine learning.

The data trainer 1310 and the data recognizer 1320 may be mounted on asingle text input device, or may be respectively mounted on separatedevices. For example, one of the data trainer 1310 and the datarecognizer 1320 may be included in the single text input device, and theother thereof may be included in the server. Also, model informationconstructed by the data trainer 1310 may be provided to the datarecognizer 1320 through wired or wireless communication, and data inputto the data recognizer 1320 may be provided to the data trainer 1310 asadditional training data.

At least one of the data trainer 1310 and the data recognizer 1320 maybe implemented as at least one software module. When at least one of thedata trainer 1310 and the data recognizer 1320 is implemented as asoftware module (or a program module including instructions), thesoftware module may be stored in a non-transitory computer-readablemedium. Also, in this case, at least one software module may be providedby an operating system (OS) or may be provided by a certain application.Alternatively, part of at least one software module may be provided byan OS, and the remaining part thereof may be provided by a certainapplication.

FIG. 16 is a block diagram of the data trainer 1310 according to someembodiments.

Referring to FIG. 16, the data trainer 1310 according to someembodiments may include a data obtainer 1310-1, a pre-processor 1310-2,a training data selector 1310-3, a model learner 1310-4, and a modelevaluator 1310-5.

The data obtainer 1310-1 may obtain data necessary for determining avoice activation and deactivation conditions or generating recommendedvoice text information. The data obtainer 1310-1 may obtain, forexample, a certain user voice and certain context information.

The pre-processor 1310-2 may pre-process the obtained data such that theobtained data is used for learning for determining the voice activationand deactivation conditions or generating the recommended voice textinformation. The pre-processor 1310-2 may process the obtained data intoa preset format such that the model learner 1310-4 to be described belowis enabled to use the obtained data for learning for determining thevoice activation and deactivation conditions and generating therecommended voice text information.

The training data selector 1310-3 may select data necessary for learningfrom among pieces of pre-processed data. The selected data may beprovided to the model learner 1310-4. The training data selector 1310-3may select data necessary for learning from among pieces ofpre-processed data according to a preset criterion for determining thevoice activation and deactivation conditions and generating therecommended voice text information. Also, the training data selector1310-3 may select data according to a preset criterion by learningperformed by the model learner 1310-4 to be described below.

The model learner 1310-4 may learn a criterion regarding whether todetermine the voice activation and deactivation conditions or generatethe recommended voice text information, based on training data. Also,the model learner 1310-4 may learn a criterion regarding which trainingdata is to be used for determining the voice activation and deactivationconditions and generating the recommended voice text information.

Also, the model learner 1310-4 may learn a criterion regarding whichtraining data is to be used for determining the voice activation anddeactivation conditions and generating the recommended voice textinformation. In this case, the data recognition model may be apre-constructed model. For example, the data recognition model may be amodel that is pre-constructed by receiving basic training data (e.g.,sample data, etc.).

The data recognition model may be constructed considering theapplication field of the recognition model, the purpose of learning, thecomputer performance of the device, or the like. The data recognitionmodel may be designed to simulate a human brain structure on a computer.The data recognition model may include a plurality of network nodeshaving weights, which simulate neurons of a human neural network. Thenetwork nodes may form a connection relationship such that the neuronssimulate the synaptic activity of the neurons that exchange signalsthrough synapses. The data recognition model may include, for example, aneural network model, or a deep learning model developed from the neuralnetwork model. In the deep learning model, a plurality of network nodesmay be located at different depths (or layers) and exchange dataaccording to a convolutional connection relationship. For example, thedata recognition model may include models such as a deep neural network(DNN), a recurrent neural network (RNN), or a bidirectional recurrentdeep neural network (BRDNN), but the present disclosure is not limitedthereto.

According to various embodiments, when there are a plurality ofpre-constructed recognition models, the model learner 1310-4 maydetermine, as a data recognition model to be learned, a data recognitionmodel having a large correlation between input training data and basictraining data. In this case, the basic training data may be previouslyclassified for each data type, and the data recognition model may bepre-constructed for each data type. For example, the basic training datamay be previously classified based on various criteria such as a regionin which the training data is generated, a time at which the trainingdata is generated, a size of the training data, a genre of the trainingdata, a creator of the training data, and a type of an object in thetraining data.

Also, the model learner 1310-4 may learn the data recognition model byusing, for example, a learning algorithm including errorback-propagation or gradient descent.

Also, the model learner 1310-4 may learn the data recognition modelthrough, for example, supervised learning using the training data as theinput value. Also, the model learner 1310-4 may self-learn the type ofdata necessary for determining the situation without separatesupervision, so as to learn the data recognition model throughunsupervised learning that finds a criterion for determining thesituation. Also, the model learner 1310-4 may learn the data recognitionmodel through, for example, reinforcement learning using a feedback onwhether a result of determining a situation according to learning iscorrect.

Also, when the data recognition model is learned, the model learner1310-4 may store the learned data recognition model. In this case, themodel learner 1310-4 may store the learned data recognition model in thememory of the electronic device including the data recognizer 1320.Alternatively, the model learner 1310-4 may store the learned datarecognition model in the memory of the electronic device including thedata recognizer 1320, which is to be described below. Alternatively, themodel learner 1310-4 may store the learned data recognition model in amemory of a server connected to the electronic device via a wired orwireless network.

In this case, the memory in which the learned data recognition model isstored may also store, for example, commands or data associated with atleast one other element of the electronic device. Also, the memory maystore software and/or programs. The programs may include, for example,kernel, middleware, application programming interface (API), and/orapplication programs (or “applications”).

When evaluation data is input to the data recognition model and arecognition result output from the evaluation data does not satisfy acertain criterion, the model evaluator 1310-5 may cause the modellearner 1310-4 to learn again. In this case, the evaluation data may bedata preset for evaluating the data recognition model.

For example, when the number or ratio of evaluation data, an analysisresult of which is inaccurate among analysis results of the learned datarecognition model for the evaluation data, exceeds a preset thresholdvalue, the model evaluator 1310-5 may evaluate that the certaincriterion is not satisfied. For example, in a case in which the certaincriterion is defined as a ratio of 2%, when the learned data recognitionmodel outputs an incorrect recognition result for more than 20 pieces ofevaluation data among a total of 1,000 pieces of evaluation data, themodel evaluator 1310-5 may evaluate that the learned data recognitionmodel is unsuitable.

When there are a plurality of learned data recognition models, the modelevaluator 1310-5 may evaluate whether each learned data recognitionmodel satisfies the certain criterion and determine a model satisfyingthe certain criterion as a final data recognition model. In this case,when there are a plurality of models satisfying the certain criterion,the model evaluator 1310-5 may determine, as the final data recognitionmodel, any one model or a certain number of models that are preset inthe descending order of evaluation scores.

At least one of the data obtainer 1310-1, the pre-processor 1310-2, thetraining data selector 1310-3, the model learner 1310-4, and the modelevaluator 1310-5 in the data trainer 1310 may be manufactured in theform of a hardware chip and mounted on the electronic device. Forexample, at least one of the data obtainer 1310-1, the pre-processor1310-2, the training data selector 1310-3, the model learner 1310-4, andthe model evaluator 1310-5 may be manufactured in the form of adedicated hardware chip for AI, or may be manufactured as part of anexisting general-purpose processor (e.g., CPU or application processor)or a dedicated graphics processor (e.g., GPU) and mounted on theabove-described various electronic devices.

Also, the data obtainer 1310-1, the pre-processor 1310-2, the trainingdata selector 1310-3, the model learner 1310-4, and the model evaluator1310-5 may be mounted on the single electronic device or may berespectively mounted on separate electronic devices. For example, someof the data obtainer 1310-1, the pre-processor 1310-2, the training dataselector 1310-3, the model learner 1310-4, and the model evaluator1310-5 may be included in the electronic device, and the others thereofmay be included in the server.

Also, at least one of the data obtainer 1310-1, the pre-processor1310-2, the training data selector 1310-3, the model learner 1310-4, andthe model evaluator 1310-5 may be implemented as a software module. Whenat least one of the data obtainer 1310-1, the pre-processor 1310-2, thetraining data selector 1310-3, the model learner 1310-4, and the modelevaluator 1310-5 is implemented as a software module (or a programmodule including instructions), the software module may be stored in anon-transitory computer-readable medium. Also, in this case, at leastone software module may be provided by an OS or may be provided by acertain application. Alternatively, part of at least one software modulemay be provided by an OS, and the remaining part thereof may be providedby a certain application.

FIG. 17 is a block diagram of the data recognizer 1320 according to someembodiments.

Referring to FIG. 17, the data recognizer 1320 according to someembodiments may include a data obtainer 1320-1, a pre-processor 1320-2,a recognition data selector 1320-3, a recognition result provider1320-4, and a model updater 1320-5.

The data obtainer 1320-1 may obtain data necessary for situationdetermination, and the pre-processor 1320-2 may pre-process the obtaineddata so as to enable the data obtained for the situation determinationto be used. The pre-processor 1320-2 may process the obtained data intoa preset format such that the recognition result provider 1320-4 usesthe obtained data for situation determination.

The recognition data selector 1320-3 may select data necessary for thesituation determination from among pieces of the pre-processed data. Theselected data may be provided to the recognition result provider 1320-4.The recognition data selector 1320-3 may select all or part of thepre-processed data according to a preset criterion for the situationdetermination. Also, the recognition data selector 1320-3 may selectdata according to a preset criterion by learning performed by the modellearner 1310-4 to be described below.

The recognition result provider 1320-4 may determine the situation byapplying the selected data to the data recognition model. Therecognition result provider 1320-4 may provide a recognition resultaccording to a data recognition purpose. The recognition result provider1320-4 may apply the selected data to the data recognition model byusing the data selected by the recognition data selector 1320-3 as aninput value. Also, the analysis result may be determined by the datarecognition model.

The model updater 1320-5 may update the data recognition model based onthe evaluation of the recognition result provided by the recognitionresult provider 1320-4. For example, the model updater 1320-5 mayprovide, to the model learner 1310-4, the recognition result provided bythe recognition result provider 1320-4, such that the model learner1310-4 updates the data recognition model.

At least one of the data obtainer 1320-1, the pre-processor 1320-2, therecognition data selector 1320-3, the recognition result provider1320-4, and the model updater 1320-5 in the data recognizer 1320 may bemanufactured in the form of a hardware chip and mounted on theelectronic device. For example, at least one of the data obtainer1320-1, the pre-processor 1320-2, the recognition data selector 1320-3,the recognition result provider 1320-4, and the model updater 1320-5 maybe manufactured in the form of a dedicated hardware chip for AI, or maybe manufactured as part of an existing general-purpose processor (e.g.,CPU or application processor) or a dedicated graphics processor (e.g.,GPU) and mounted on the above-described various electronic devices.

Also, the data obtainer 1320-1, the pre-processor 1320-2, therecognition data selector 1320-3, the recognition result provider1320-4, and the model updater 1320-5 may be mounted on the singleelectronic device or may be respectively mounted on separate electronicdevices. For example, some of the data obtainer 1320-1, thepre-processor 1320-2, the recognition data selector 1320-3, therecognition result provider 1320-4, and the model updater 1320-5 may beincluded in the electronic device, and the others thereof may beincluded in the server.

Also, at least one of the data obtainer 1320-1, the pre-processor1320-2, the recognition data selector 1320-3, the recognition resultprovider 1320-4, and the model updater 1320-5 may be implemented as asoftware module. When at least one of the data obtainer 1320-1, thepre-processor 1320-2, the recognition data selector 1320-3, therecognition result provider 1320-4, and the model updater 1320-5 isimplemented as a software module (or a program module includinginstructions), the software module may be stored in a non-transitorycomputer-readable medium. Also, in this case, at least one softwaremodule may be provided by an OS or may be provided by a certainapplication. Alternatively, part of at least one software module may beprovided by an OS, and the remaining part thereof may be provided by acertain application.

FIG. 18 is a diagram illustrating an example in which a text inputdevice 1000 and a server 2000 interoperate with each other to train andrecognize data, according to some embodiments.

Referring to FIG. 18, the server 2000 may learn a criterion fordetermining voice activation and deactivation conditions and generatingrecommended voice text information, and the text input device 1000 maydetermine the determination of the voice activation and deactivationconditions and the generation of the recommended voice text information,based on a result of learning performed by the server 2000.

In this case, a model learner 2340 of the server 2000 may perform thefunction of the data trainer 1310 illustrated in FIG. 16. A modellearner 2340 of the server 2000 may learn a criterion regarding whichdata is to be used for determining the voice activation and deactivationconditions and generating the recommended voice text information. Also,the model learner 2340 of the server 2000 may learn a criterion fordetermining the voice activation and deactivation conditions andgenerating the recommended voice text information by using data. Themodel learner 2340 may learn a criterion for determining the voiceactivation and deactivation conditions and generating the recommendedvoice text information by obtaining data to be used for learning andapplying the obtained data to a data recognition model to be describedbelow.

Also, the recognition result provider 1320-4 of the text input device1000 may determine voice activation and deactivation conditions andgenerate recommended voice text information by applying the dataselected by the recognition data selector 1320-3 to a data recognitionmodel generated by the server 2000. For example, the recognition resultprovider 1320-4 may transmit the data selected by the recognition dataselector 1320-3 to the server 2000, and the server 2000 may make arequest for determining the voice activation and deactivation conditionsand generating the recommended voice text information by applying thedata selected by the recognition data selector 1320-3 to the datarecognition model. Also, the recognition result provider 1320-4 mayreceive, from the server 2000, information about the determination ofthe voice activation and deactivation conditions and the generation ofthe recommended voice text information, which are determined by theserver 2000.

Alternatively, the recognition result provider 1320-4 of the text inputdevice 1000 may receive, from the server 2000, the data recognitionmodel generated by the server 2000 and may use the received datarecognition model to determine the voice activation and deactivationconditions and generate the recommended voice text information. In thiscase, the recognition result provider 1320-4 of the text input device1000 may determine the voice activation and deactivation conditions andgenerate the recommended voice text information by applying the dataselected by the recognition data selector 1320-3 to the data recognitionmodel received from the server 2000.

Also, the text input device 1000 and the server 2000 may effectivelydistribute and perform operations for the learning of the datarecognition model and the data recognition. In this manner, dataprocessing may be efficiently performed so as to provide a servicematching a user's intention and a user's privacy may be effectivelyprotected.

Some embodiments of the present disclosure may be implemented as asoftware program that includes instructions stored in acomputer-readable storage medium.

For example, a computer is a device capable of calling the storedinstructions from the storage medium and performing the operations ofthe embodiments according to the called instructions. The computer mayinclude the device according to the embodiments or an external servercommunicatively connected to the device.

The computer-readable storage medium may be provided in the form of anon-transitory storage medium. The term “non-transitory” means that thestorage media does not include a signal or current and is tangible, anddoes not distinguish between semi-permanent and temporary storage ofdata in the storage medium. For example, the non-transitory storagemedium may include not only a non-transitory readable storage mediumsuch as compact disc (CD), digital versatile disc (DVD), hard disk,Blu-ray disk, universal serial bus (USB), internal memory, memory card,ROM, or RAM, but also a temporary storage medium such as register,cache, or buffer.

Also, the methods according to the embodiments may be provided as acomputer program product.

The computer program product may include a software program, acomputer-readable storage medium on which the software program isstored, or a product traded between a seller and a buyer.

For example, the computer program product may include a product (e.g.,downloadable app) of a software program form that is distributedelectronically through a device manufacturer or an electronic market(e.g., Google Play Store, App Store, etc.). For electronic distribution,at least part of the software program may be stored in a storage mediumor may be temporarily generated. In this case, the storage medium may bea server of a manufacturer or an electronic market, or a storage mediumof a relay server.

1. An electronic device comprising: a display; a microphone; a memoryconfigured to store at least one instruction; and at least one processorwhich is, by executing the at least one instruction, configured to:activate a voice input function based on a user input, based on thevoice input function being activated, display, on the display, a graphicrepresentation for indicating that the voice input function isactivated, while the graphic representation indicating that the voiceinput function is activated is displayed, display, on the display, atext display area for displaying text inputted by a plurality of userinput methods and a keyboard input interface including a plurality ofvirtual buttons respectively corresponding to a plurality of alphabetcharacters which are displayed differently according to a selectedlanguage for receiving a user keyboard input, the plurality of userinput methods including a user keyboard input method and a user voiceinput method, receive, via the plurality of virtual buttons included inthe keyboard input interface, the user keyboard input corresponding to afirst text while the graphic representation indicating that the voiceinput function is activated is displayed, display, on the display, thefirst text in the text display area based on the receiving the userkeyboard input together with the graphic representation indicating thatthe voice input function is activated, automatically receive, via themicrophone, a user voice input corresponding to a second text without anadditional user input for activating the voice input function of theelectronic device while both of the keyboard input interface includingthe plurality of virtual buttons respectively corresponding to theplurality of alphabet characters, and the graphic representationindicating that the voice input function is activated are displayed onthe display, and display, on the display, the second text next to thefirst text in the text display area based on the receiving the uservoice input, wherein a display order of the first text obtained from theuser keyboard input and the second text obtained from the user voiceinput is changeable based at least on an input order of the userkeyboard input and the user voice input while the graphic representationindicating that the voice input function is activated is displayed onthe display.
 2. The electronic device of claim 1, wherein, based on thereceiving the user voice input, a recommended word is provided and therecommended word is selected to be displayed in the text display area.3. The electronic device of claim 1, wherein the user input comprisesuttering a voice by a user that is detected by the microphone.
 4. Theelectronic device of claim 1, wherein the user input comprises changinga position of the electronic device, and wherein the changing of theposition is detected by at least one of an accelerometer or a gyroscope,of the electronic device.
 5. The electronic device of claim 4, whereinthe changed position of the electronic device is suitable for a user toview the display of the electronic device.
 6. The electronic device ofclaim 1, wherein the user input comprises a user's contact with theelectronic device.
 7. The electronic device of claim 1, wherein thegraphic representation comprises an icon related to the voice inputfunction.
 8. The electronic device of claim 1, wherein, based on thereceiving the user voice input, a recommended word is provided betweenthe keyboard input interface and the text display area.
 9. Theelectronic device of claim 1, wherein, based on the receiving the uservoice input, a recommended word is provided based on usage historyinformation of a user.
 10. The electronic device of claim 9, wherein theusage history information comprises a history of a function executed inan application.
 11. The electronic device of claim 9, wherein the usagehistory information comprises a history of a text messaging of the user.12. The electronic device of claim 9, wherein the usage historyinformation comprises a pattern of words being input by the user. 13.The electronic device of claim 1, wherein the at least one processor isfurther configured to: transmit the user voice input to a server for atleast one of an analysis or a processing.
 14. The electronic device ofclaim 1, wherein, based on the receiving the user voice input, arecommended word is provided by using a model trained by at least one ofa server or the electronic device.
 15. The electronic device of claim 1,wherein, based on the receiving the user voice input, a recommended wordis provided by using a model received from a server.
 16. The electronicdevice of claim 1, wherein, based on the receiving the user voice input,a recommended word is provided and the recommended word is related to atleast one of the first text or the second text.
 17. A method performedby an electronic device, the method comprising: activating a voice inputfunction based on a user input; based on the voice input function beingactivated, displaying, on a display of the electronic device, a graphicrepresentation for indicating that the voice input function isactivated; while the graphic representation indicating that the voiceinput function is activated is displayed, displaying, on the display, atext display area for displaying text inputted by a plurality of userinput methods and a keyboard input interface including a plurality ofvirtual buttons respectively corresponding to a plurality of alphabetcharacters which are displayed differently according to a selectedlanguage for receiving a user keyboard input, the plurality of userinput methods including a user keyboard input method and a user voiceinput method; receiving, via the plurality of virtual buttons includedin the keyboard input interface, the user keyboard input correspondingto a first text while the graphic representation indicating that thevoice input function is activated is displayed; displaying, on thedisplay, the first text in the text display area based on the receivingthe user keyboard input together with the graphic representationindicating that the voice input function is activated; automaticallyreceiving, via a microphone, a user voice input corresponding to asecond text without an additional user input for activating the voiceinput function of the electronic device while both of the keyboard inputinterface including the plurality of virtual buttons respectivelycorresponding to the plurality of alphabet characters, and the graphicrepresentation indicating that the voice input function is activated aredisplayed on the display; and displaying, on the display, the secondtext next to the first text in the text display area based on thereceiving the user voice input, wherein a display order of the firsttext obtained from the user keyboard input and the second text obtainedfrom the user voice input is changeable based at least on an input orderof the user keyboard input and the user voice input while the graphicrepresentation indicating that the voice input function is activated isdisplayed on the display.
 18. A non-transitory computer-readable mediumhaving instructions recorded thereon which, when executed by at leastone processor, cause the at least one processor to execute the method ofclaim 17.